Software Quality Assurance

CHAPTER OVERVIEW AND COMMENTS

This chapter provides an introduction to software quality assurance (SQA). It is important to have the students understand that software quality work begins before the testing phase and continues after the software is delivered. The role of metrics in software management is reinforced in this chapter.

7.1 Quality Concepts

An important concept in this section is that controlling variation among products is what quality assurance work is all about. Software engineers are concerned with controlling the variation in their processes, resource expenditures, and the quality attributes of the end products. The definitions of many quality concepts appear in this section. Students need to be familiar with these definitions, since their use in software quality work does not always match their use in casual conversation. Students also need to be made aware that customer satisfaction is every bit as important to modern quality work as is quality of design and quality of conformance.

 

7.2 The Quality Movement

Total quality management (TQM) is described in this section. TQM stresses continuous process improvement and can be applied to software development. The most important point made in the section is that until a visible, repeatable, and measurable process is created not much can be done to improve quality.

 

7.3 Software Quality Assurance

This section describes software quality as conformance to explicitly stated requirements and standards, as well as implicit characteristics that customers assume will be present in any professionally developed software. The SQA group must look at software from the customer's perspective, as well as assessing its technical merits. The activities performed by the SQA group involve quality planning, oversight, record keeping, analysis and reporting. SQA plans are discussed in more detail later in this chapter.

 

7.4 Software Reviews

This section describes the purposes of conducting software reviews. It is important to point out to students that any work product (including documents) should be reviewed. Students are usually impressed by the fact that conducting timely reviews of all work products can often eliminate 80% of the defects before any testing is conducted. This message often needs to be carried to managers in the field, whose impatience to generate code sometimes makes them reluctant to spend time on reviews.

 

7.5 Formal Technical Reviews

The mechanics of conducting a formal technical review (FTR) are described in this section. Students should pay particular attention to the point that it is the work product that is being reviewed not the producer. Encouraging the students to conduct formal reviews of their development projects is a good way to make this section more meaningful. Requiring students to generate review summary reports and issues lists also helps to reinforce the importance of the review activities.

 

7.6 Formal Approaches to SQA

This section merely introduces the concept of formal methods in software engineering. More comprehensive discussions of formal specification techniques and formal verification of software appear Chapters 25 and 26.

 

7.7 Statistical Quality Assurance

Statistical quality assurance is beyond the scope of this text. However, this section does contain a high level description of the process and gives examples of metrics that might be used in this type of work. The key points to emphasize to students are that each defect needs to be traced to its cause and that defect causes having the greatest impact on the success of the project must be addressed first.

 

7.8 Software Reliability

Software reliability is discussed in this section. It is important to have the students distinguish between software consistency (repeatability of results) and reliability (probability of failure free operation for a specified time period). Students should be made aware of the arguments for and against applying hardware reliability theory to software (e.g. a key point is that, unlike hardware, software does not wear out so that failures are likely to be caused by design defects). It is also important for students to be able to make a distinction between software safety (identifying and assessing the impact of potential hazards) and software reliability.

 

7.9 A Software Quality Assurance Approach

This section describes the use of poka-yoke devices as mechanisms that lead to the prevention of potential quality problems or the rapid detection of quality problems introduced into a work product. Examples of poka-yoke devices are given, but students will need to see others (a web reference is given in the text).

 

7.10 The ISO 9000 Quality Standards

The ISO 9000 quality standard is discussed in this section as an example of quality model that is based on the assessment of quality of the individual processes used in the enterprise as a whole. ISO 9001 is described as the quality assurance standard that contains 20 requirements that must be present in any software quality assurance system.

 

7.11 The SQA Plan

The major sections of a SQA plan are described in this section. It would advisable to have students write a SQA plan for one of their own projects sometime during the course. This will be a difficult task for them. It may be advisable to have the students look at the material in Chapters 17-19 (testing and technical metrics) before beginning this assignment.

In addition to the review checklists contained within the SEPA Web site, I have also included a small sampler in the special section that follows.

 

Review Checklists

Formal technical reviews can be conducted during each step in the software engineering process. In this section, we present a brief checklist that can be used to assess products that are derived as part of software development. The checklists are not intended to be comprehensive, but rather to provide a point of departure for each review.

 

System Engineering. The system specification allocates function and performance to many system elements. Therefore, the system review involves many constituencies that may each focus on their own area of concern. Software engineering and hardware engineering groups focus on software and hardware allocation, respectively. Quality assurance assesses system level validation requirements and field service examines the requirements for diagnostics. Once all reviews are conducted, a larger review meeting, with representatives from each constituency, is conducted to ensure early communication of concerns. The following checklist covers some of the more important areas of concern.

 

1. Are major functions defined in a bounded and unambiguous fashion?

2. Are interfaces between system elements defined?

3. Have performance bounds been established for the system as a whole and for each element?

4. Are design constraints established for each element?

5. Has the best alternative been selected?

6. Is the solution technologically feasible?

7. Has a mechanism for system validation and verification been established?

8. Is there consistency among all system elements?

 

Software Project Planning. Software project planning develops estimates for resources, cost and schedule based on the software allocation established as part of the system engineering activity. Like any estimation process, software project planning is inherently risky. The review of the Software Project Plan establishes the degree of risk. The following checklist is applicable.

1. Is software scope unambiguously defined and bounded?

2. Is terminology clear?

3. Are resources adequate for scope?

4. Are resources readily available?

5. Have risks in all important categories been defined.

6. Is a risk management plan in place?

7. Are tasks properly defined and sequenced? Is parallelism reasonable given available resources?

8. Is the basis for cost estimation reasonable? Has the cost estimate been developed using two independent methods?

9. Have historical productivity and quality data been used?

10. Have differences in estimates been reconciled?

11. Are pre-established budgets and deadlines realistic?

12. Is the schedule consistent?

 

Software Requirements Analysis. Reviews for software requirements analysis focus on traceability to system requirements and consistency and correctness of the analysis model. A number of FTRs are conducted for the requirements of a large system and may be augmented by reviews and evaluation of prototypes as well as customer meetings. The following topics are considered during FTRs for analysis:

1. Is information domain analysis complete, consistent and accurate?

2. Is problem partitioning complete?

3. Are external and internal interfaces properly defined?

4. Does the data model properly reflect data objects, their attributes and relationships.

5. Are all requirements traceable to system level?

6. Has prototyping been conducted for the user/customer?

7. Is performance achievable within the constraints imposed by other system elements?

8. Are requirements consistent with schedule, resources and budget?

9. Are validation criteria complete?

 

Software Design. Reviews for software design focus on data design, architectural design and procedural design. In general, two types of design reviews are conducted. The preliminary design review assesses the translation of requirements to the design of data and architecture. The second review, often called a design walkthrough, concentrates on the procedural correctness of algorithms as they are implemented within program modules. The following checklists are useful for each review:

 

Preliminary design review

1. Are software requirements reflected in the software architecture?

2. Is effective modularity achieved? Are modules functionally independent?

3. Is the program architecture factored?

4. Are interfaces defined for modules and external system elements?

5. Is the data structure consistent with information domain?

6. Is data structure consistent with software requirements?

7. Has maintainability considered?

8. Have quality factors (section 17.1.1) been explicitly assessed?

 

Design walkthrough

1. Does the algorithm accomplishes desired function?

2. Is the algorithm logically correct?

3. Is the interface consistent with architectural design?

4. Is the logical complexity reasonable?

5. Have error handling and "anti-bugging" been specified?

6. Are local data structures properly defined?

7. Are structured programming constructs used throughout?

8. Is design detail amenable to implementation language?

9. Which are used: operating system or language dependent features?

10. Is compound or inverse logic used?

11. Has maintainability considered?

 

Coding. Although coding is a mechanistic outgrowth of procedural design, errors can be introduced as the design is translated into a programming language. This is particularly true if the programming language does not directly support data and control structures represented in the design. A code walkthrough can be an effective means for uncovering these translation errors. The checklist that follows assumes that a design walkthrough has been conducted and that algorithm correctness has been established as part of the design FTR.

1. Has the design properly been translated into code? [The results of the procedural design should be available during this review.]

2. Are there misspellings and typos?

3. Has proper use of language conventions been made?

4. Is there compliance with coding standards for language style, comments, module prologue?

5. Are there incorrect or ambiguous comments?

6. Are data types and data declaration proper?

7. Are physical constants correct?

8. Have all items on the design walkthrough checklist been re-applied (as required)?

 

Software Testing. Software testing is a quality assurance activity in it own right. Therefore, it may seem odd to discuss reviews for testing. However, the completeness and effectiveness of testing can be dramatically improved by critically assessing any test plans and procedures that have been created. In the next two Chapters, test case design techniques and testing strategies are discussed in detail.

 

Test plan

1. Have major test phases properly been identified and sequenced?

2. Has traceability to validation criteria/requirements been established as part of software requirements analysis?

3. Are major functions demonstrated early?

4. Is the test plan consistent with overall project plan?

5. Has a test schedule been explicitly defined?

6. Are test resources and tools identified and available?

7. Has a test record keeping mechanism been established?

8. Have test drivers and stubs been identified and has work to develop them been scheduled?

9. Has stress testing for software been specified?

Test procedure

1. Have both white and black box tests been specified?

2. Have all independent logic paths been tested?

3. Have test cases been identified and listed with expected results?

4. Is error-handling to be tested?

5. Are boundary values to be tested?

6. Are timing and performance to be tested?

7. Has acceptable variation from expected results been specified?

In addition to the formal technical reviews and review checklists noted above, reviews (with corresponding checklists) can be conducted to assess the readiness of field service mechanisms for product software; to evaluate the completeness and effectiveness of training; to assess the quality of user and technical documentation, and to investigate the applicability and availability of software tools.

 

Maintenance. The review checklists for software development are equally valid for the software maintenance phase. In addition to all of the questions posed in the checklists, the following special considerations should be kept in mind:

1. Have side effects associated with change been considered?

2. Has the request for change been documented, evaluated and approved?

3. Has the change, once made, been documented and reported to interested parties?

4. Have appropriate FTRs been conducted?

5. Has a final acceptance review been conducted to ensure that all software has been properly updated, tested and replaced?

 

PROBLEMS AND POINTS TO PONDER

 

7.1. We look for variation in the traceability from requirements to design and design to code. We assess the variation in the form and content of work products. We look for variation in the process — repeatable process is a goal. We look for variation in expected and actual results derived from software testing.

7.2. In reality, if we define quality as "conformance to requirements," and requirements are dynamic (keep changing), the definition of quality will be dynamic and an assessment of quality will be difficult.

7.3. Quality focuses on the software's conformance to explicit and implicit requirements. Reliability focuses on the ability of software to function correctly as a function of time or some other quantity. Safety considers the risks associated with failure of a computer-based system that is controlled by software. In most cases an assessment of quality considers many factors that are qualitative in nature. Assessment of reliability and to some extent safety is more quantitative, relying on statistical models of past events that are coupled with software characteristics in an attempt to predict future operation of a program.

7.4. Yes. It is possible for a program to conform to all explicit functional and performance requirements at a given instant, yet have errors that cause degradation that ultimately causes the program to fail.

7.5. Absolutely. Many programs have been patched together or otherwise "kludged up" so that they work, yet these program exhibit very low quality if measured by most of the criteria described in Figure 17.1.

7.6. There is often a natural "tension" that exists between these two groups. The reason is simple: if the SQA group takes on the role of the "watch dog," flagging quality problems and high-lighting shortcomings in the developed software, it is only normal that this would not be embraced with open arms by the software engineering group. As long as the tension does not degenerate into hostility, there is no problem. It is important to note, however, that a software engineering organization should work to eliminate this tension by encouraging a team approach that has development and QA people working together toward a common goal—high quality software.

7.7. Institute formal technical reviews. After these are working smoothly, any of a number of SQA activities might be implemented: change control and SCM (see Chapter 9); comprehensive testing methodology (see Chapters 16 and 17); SQA audits of documentation and related software. Also software metrics can help (Chapters 4, 18, and 23)

7.8. Any countable measure that indicates the factors noted in Chapter 18 are candidates. For example, maintainability as measured by mean-time-to-change; portability as measured by an index that indicates conformance to language standard; complexity as measured by McCabe's metric, and so on.

7.9. Typically, an unprepared reviewer will be reading the product materials while everyone else is conducting the review; will be especially quiet throughout the review; will have made no annotation on product materials; will make only very general comments; will focus solely on one part (the part he/she read) of the product materials. As a review leader, you should ask if the person has had time to prepare. If most reviewers have not prepared, the review should be cancelled and rescheduled.

7.10. Assessing style is tricky and can lead to bad feelings if a reviewer is not careful when he/she makes comments concerning style. If the producer gets the feeling that the reviewer is saying, "Do it like I do," it is likely that some resentment will arise. In general, the review should focus on correctness.

7.12. You might suggest that students create a spreadsheet model using a table like Table 7.2 and the equations discussed in Section 7.7 to do the necessary computation.

7.13 The May, 1995 and January, 1996 issues of IEEE Software have useful articles.

7.14. For hardware the MTBF concept is based on statistical error data that occurs due to physical wear in a product. In general, when a failure does occur in hardware, the failed part is replaced with a spare. However, when an error occurs for software, a design change is made to correct it. The change may create side effects that generate other errors. Therefore, the statistical validity of MTBF for software is suspect.

7.15. Classic examples include aircraft avionics systems, control systems for nuclear power plants, software contained in sophisticated medical instrumentation (e.g., CAT scanners or MRI devices) control systems for trains or subway systems; elevator control systems.

7.17 device: Develop a simple parsing algorithm that will detect common errors in e-mail addresses (e.g., bad characters, double dots)

detection device: recognize an e-mail message that has no content (e.g., e-mail text is null)

7.18 See the SEPA Web site for links to ISO sites.